One of the great things about Magic Leap One mixed reality technology is that it allows developers to place virtual content onto real-world objects and surfaces, those who were scanned and meshed by the system.
For example, you can take a virtual dinosaur and put it on the table in your living room, behind a chair and even within objects with depth like a box (if its inner side was scanned and meshed properly.
That being said, developers might take advantage of another possibility, which is deforming surfaces or creating an effect of a deformed surface.
I think giving some examples can better demonstrate what I mean.
Simulating depth inside surfaces
The screenshot above taken from the game Defend It! AR. The developers put a masked 3D content located behind the wall’s focal plane along the z-axis to create the illusion of depth behind the wall. It made a nice natural-looking cutout that indeed gives the impressions of a broken wall. He then spawned enemies from within that area, coming outside to the real world.
This is the same effect that was used in the game ARrrrrgh, where the user gets the effect of digging into the floor and planting a treasure chest. It was demonstrated also in AMON.
Some game used portals. Making temporary holes in surfaces that open up, something comes from them and then they are closed. Sometimes there is a texture or a 3D extension of the portal to give the illusion of depth like there is something on the other side of the portal. Kind of like a flat wormhole that once you go through it, you find yourself in another place. This effect was used brilliantly in Alice in Wonderland AR Quest.
You can use these type of effects in your Mixed Reality apps to add another visual stimulation layer to help deliver a more compelling and immersive experience or just mimic some real-world interactions inherited brought into the mixed reality world in a virtual form (e.g. digging under the floor with a shovel)
Surface Deformation
I haven’t dug very deep into how this can be achieved using the Magic Leap One, but it doesn’t seem too complicated to do.
The idea is to give the illusion of surfaces deforming, changing their shape. This should work well with surfaced that have unified texture, To make it possible, the developer needs to get a texture sample of the surface. Having said that, you can create something coming out of the surface even without full matching texture or color.
So imagine throwing a ball onto the wall in your room and seeing the wall reacting like a soft sponge, with the ball partially penetrating the wall and then thrown back. Another example, virtually extending part of your table upwards, or making look like a face is morphing out of the wall. If there is a fire in a game, you can make some bubbling effect on surfaces as a high-temperature cue. You can really exaggerate and make some really scary stuff, which is great if you are making a horror story-telling experience 🙂
The thing is that some of the virtual objects that you use in your app will have their own physical properties. Imagine you have two balls, one made of metal and one is like a balloon. Now, you can apply physics to the ball so if it collides with the sofa it either stays on it when you drop it or react like a ballon and moves up into the air again. The thing is that if you don’t add any surface deformation, it will look like the surface isn’t reacting to that collision with the ball. So if it was a sofa and you threw the metal ball on it, you should add some surface deformation effect so it gives the feeling that a weight is applied to the surface.
Using surface deformation can make mixed reality experience feel more realistic, making the virtual object appear a more integral part of the real world. By the way, combining this with masking can lead to an even better experience and sometimes even required. For example, imagine a metal ball on a soft surface, part of it sinks into the fabric, so you’ll need to mask some of the bottom parts to make it appear realistic.
Applied texture
Another option for surface deformation is changing or adding texture. This doesn’t require any 3D deformation.
You can just apply a PNG texture to surfaces to give the effect of something. For example, if you shoot the wall, you can put a texture of a hole in the wall. If you are breaking a glass, you can put a texture of a broken glass (glass maybe isn’t a good idea because it can’t be scanned by the Magic Leap One from all I know). If there is fire, after the use put it off, you can leave some burn surface texture. If the user hits the wall with something, you can put cracks texture on the wall. If you shoot objects and hit the wall, you can put dust texture on surfaces below the area on the wall where the bullet hit, etc.
You can even use animation or add reflecting objects to imply wet surfaces. There are plenty of other examples of course, but I assume that you get the idea based on these examples.
Painting Surfaces
Another way to spice things up is to apply different shaders or textures to a large surface area or objects. You can use is to design different looking levels for your game, for example, giving the surface Tron-like visuals, giving your game or app a sci-fi touch. There isn’t any strict rule that forbids you to retouch reality. In fact, it can lead to some really unique, stimulating and visually appealing results.
You don’t need to make that permanent. You can paint surfaces temporarily to imply change, like if something is going to come out behind the wall. Give a strong visual cue for the user doing so.
You can also paint the environment as part of the experience itself. For example, making a game like Splatoon where you fight against another local player who paints the largest area under 1 minute. The paint will exist throughout the game session then reset.
Summary
There are probably other uses that I didn’t mention here and if they pop up I will add them here later on. Of course everything I mentioned here is totally optional. These type of surface deformation and effects might work well for some apps and games while not so well for others.
Using these techniques, you can spice up your mixed reality experience. It’s important to note that I don’t have the Magic Leap One nor any other Mixed Reality headset in that regard, so I can’t tell you if it will work well or not.or how to do it.
If you are trying one of those techniques in your mixed reality app, please PM and let me know how it worked out for you, I really want to see how they turn out to be.
I hope you enjoy reading this article. If you do, please don’t forget to share it. Thank you.